Papers with Intermediate Layer Distillation
Revisiting Intermediate Layer Distillation for Compressing Language Models: An Overfitting Perspective (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation (KD) are prone to overfitting to training datasets . recent advances in NLP have shown that using PLMs such as BERT and RoBERTa on downstream tasks is effective. |
| Approach: | They propose a consistency-regularized knowledge distillation method which mitigates overfitting of existing methods. |
| Outcome: | The proposed method outperforms existing methods on the GLUE benchmark and synthetic datasets. |
CILDA: Contrastive Data Augmentation Using Intermediate Layer Knowledge Distillation (2022.coling-1)
Copied to clipboard
Md Akmal Haidar, Mehdi Rezagholizadeh, Abbas Ghaddar, Khalil Bibi, Phillippe Langlais, Pascal Poupart
| Challenge: | Knowledge distillation (KD) is an efficient framework for compressing large-scale pre-trained language models. |
| Approach: | They propose a data augmentation technique tailored for knowledge distillation based on contrastive loss to improve masked adversarial data augmented by intermediate layer matching. |
| Outcome: | The proposed technique outperforms state-of-the-art methods on the GLUE benchmark and in an out-of domain evaluation. |
Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods underestimate the importance of utilizing the teacher's discriminative classifier and face challenges in establishing proper layer mappings. |
| Approach: | They propose to reuse pre-trained teacher classifiers to improve student performance . they use projectors to match hidden size of the teacher model to student . |
| Outcome: | The proposed method outperforms existing methods on 97.7% of the teacher BERT base without additional trainable parameters. |